{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(hyper-params)=\n",
    "# Hyperparameter tuning optimization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MLRun supports iterative tasks for automatic and distributed execution of many tasks with variable parameters (hyperparams). Iterative tasks can be distributed across multiple containers. They can be used for:\n",
    "* Parallel loading and preparation of many data objects\n",
    "* Model training with different parameter sets and/or algorithms\n",
    "* Parallel testing with many test vector options\n",
    "* AutoML\n",
    "\n",
    "MLRun iterations can be viewed as child runs under the main task/run. Each child run gets a set of parameters that are computed/selected from the input hyperparameters based on the chosen strategy ([Grid](#grid-search-default), [List](#list-search), [Random](#random-search) or [Custom](#custom-iterator)).\n",
    "\n",
    "The different iterations can run in parallel over multiple containers (using Dask or Nuclio runtimes, which manage the workers). Read more in [Parallel execution over containers](#parallel-execution-over-containers).\n",
    "\n",
    "The hyperparameters and options are specified in the `task` or the {py:meth}`~mlrun.runtimes.BaseRuntime.run` command \n",
    "through the `hyperparams` (for hyperparam values) and `hyper_param_options` (for \n",
    "{py:class}`~mlrun.model.HyperParamOptions`) properties. See the examples below. \n",
    "\n",
    "The hyperparams are specified as a struct of `key: list` values. The values can be of any type (int, string, float, ..). \n",
    "The lists are used to compute the parameter combinations using one of the \n",
    "following strategies: \n",
    "- [Grid search](#grid-search-default) (`grid`) &mdash; running all the parameter combinations. The `key: list` values structure is similar to: \n",
    "  ` { \"p1\": [1,2], \"p2\": [2,4] }`<br>\n",
    "   The result is the four iterations with all the combinations of p1 and p2. \n",
    "   Hyperparameters can also be loaded directly from a JSON file (specify `param_file` in {py:class}`~mlrun.model.HyperParamOptions`).\n",
    "- [Random](#random-search) (`random`) &mdash; running a sampled set from all the parameter combinations. Hyperparameters can       also be loaded directly from a JSON file, the same as `grid`.\n",
    "- [List](#list-search) (`list`) &mdash; running the first parameter from each list followed by the second from each list and so on. **All the lists must be of equal length**. Hyperparameters can also be loaded directly from a JSON or CSV file containing a list of the iterations to be executed. Example JSON: `{\"p1\": [1], \"p2\": [10]}` (specify `param_file` in {py:class}`~mlrun.model.HyperParamOptions`).\n",
    "\n",
    "You can specify a selection criteria to select the best run among the different child runs by setting the `selector` option. This marks the selected result as the parent (iteration 0) result, and marks the best result in the user interface.\n",
    "\n",
    "You can also specify the `stop_condition` to stop the execution of child runs when some criteria, based on the returned results, is met (for example `stop_condition=\"accuracy>=0.9\"`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**In this section**\n",
    "- [Basic code](#basic-code)\n",
    "- [Review the results](#review-the-results)\n",
    "- [Examples](#examples)\n",
    "- [Parallel execution over containers](#parallel-execution-over-containers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic code\n",
    "\n",
    "Here's a basic example of running multiple jobs in parallel for **hyperparameters tuning**, selecting the best run with respect to the `max accuracy`. \n",
    "\n",
    "Run the hyperparameters tuning job by using the keywords arguments: \n",
    "\n",
    "* `hyperparams` for the hyperparameters options and values of choice.\n",
    "* `selector` for specifying how to select the best model.\n",
    "\n",
    "```python\n",
    "hp_tuning_run = project.run_function(\n",
    "    \"trainer\", \n",
    "    inputs={\"dataset\": gen_data_run.outputs[\"dataset\"]}, \n",
    "    hyperparams={\n",
    "        \"n_estimators\": [100, 500, 1000], \n",
    "        \"max_depth\": [5, 15, 30]\n",
    "    }, \n",
    "    selector=\"max.accuracy\", \n",
    "    local=True\n",
    ")\n",
    "```\n",
    "\n",
    "The returned run object in this case represents the `parent` (and the **best** result). You can also access the \n",
    "individual child runs (called iterations) in the MLRun UI."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Review the results\n",
    "\n",
    "When running a hyperparam job, the job `results` tab shows the list and marks the best run:\n",
    "\n",
    "<img src=\"./_static/images/hyperparam-results.png\" alt=\"results\" width=\"800\"/>\n",
    "\n",
    "You can also view results by printing the artifact `iteration_results`:\n",
    "\n",
    "```hp_tuning_run.artifact(\"iteration_results\").as_df()```\n",
    "\n",
    "MLRun also generates a `parallel coordinates plot` for the run, you can view it in the MLRun UI.\n",
    "\n",
    "![parallel_coordinates](./_static/images/parallel-coordinates.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examples\n",
    "\n",
    "**Base dummy function:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2021-10-23 12:47:39,982 [warning] Failed resolving version info. Ignoring and using defaults\n",
      "> 2021-10-23 12:47:43,488 [warning] Unable to parse server or client version. Assuming compatible: {'server_version': '0.8.0-rc7', 'client_version': 'unstable'}\n"
     ]
    }
   ],
   "source": [
    "import mlrun"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def hyper_func(context, p1, p2):\n",
    "    print(f\"p1={p1}, p2={p2}, result={p1 * p2}\")\n",
    "    context.log_result(\"multiplier\", p1 * p2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Grid search (default)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2021-10-23 12:47:43,505 [info] starting run grid-demo uid=29c9083db6774e5096a97c9b6b6c8e93 DB=http://mlrun-api:8080\n",
      "p1=2, p2=10, result=20\n",
      "p1=4, p2=10, result=40\n",
      "p1=1, p2=10, result=10\n",
      "p1=2, p2=20, result=40\n",
      "p1=4, p2=20, result=80\n",
      "p1=1, p2=20, result=20\n",
      "> 2021-10-23 12:47:44,851 [info] best iteration=5, used criteria max.multiplier\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #4EC64B;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #4EC64B;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #4EC64B;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>default</td>\n",
       "      <td><div title=\"29c9083db6774e5096a97c9b6b6c8e93\"><a href=\"https://dashboard.default-tenant.app.yh38.iguazio-cd2.com/mlprojects/default/jobs/monitor/29c9083db6774e5096a97c9b6b6c8e93/overview\" target=\"_blank\" >...6b6c8e93</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Oct 23 12:47:43</td>\n",
       "      <td>completed</td>\n",
       "      <td>grid-demo</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=admin</div><div class=\"dictlist\">kind=handler</div><div class=\"dictlist\">owner=admin</div></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">best_iteration=5</div><div class=\"dictlist\">multiplier=80</div></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultcfdef132\" title=\"files/v3io/projects/default/artifacts/iteration_results.csv\">iteration_results</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"resultcfdef132-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"resultcfdef132-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"resultcfdef132\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"resultcfdef132-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<b> > to track results use the .show() or .logs() methods  or <a href=\"https://dashboard.default-tenant.app.yh38.iguazio-cd2.com/mlprojects/default/jobs/monitor/29c9083db6774e5096a97c9b6b6c8e93/overview\" target=\"_blank\">click here</a> to open in UI</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2021-10-23 12:47:45,071 [info] run executed, status=completed\n"
     ]
    }
   ],
   "source": [
    "grid_params = {\"p1\": [2, 4, 1], \"p2\": [10, 20]}\n",
    "task = mlrun.new_task(\"grid-demo\").with_hyper_params(\n",
    "    grid_params, selector=\"max.multiplier\"\n",
    ")\n",
    "run = mlrun.new_function().run(task, handler=hyper_func)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**UI Screenshot:**\n",
    "<br><br>\n",
    "<img src=\"_static/images/hyper-params.png\" alt=\"hyper-params\" width=\"800\"/>\n",
    "\n",
    "\n",
    "### Random Search\n",
    "MLRun chooses random parameter combinations. Limit the number of combinations using the `max_iterations` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2021-10-23 12:47:45,077 [info] starting run random-demo uid=cac368c7fc33455f97ca806e5c7abf2f DB=http://mlrun-api:8080\n",
      "p1=2, p2=20, result=40\n",
      "p1=4, p2=10, result=40\n",
      "p1=3, p2=10, result=30\n",
      "p1=3, p2=20, result=60\n",
      "> 2021-10-23 12:47:45,966 [info] best iteration=4, used criteria max.multiplier\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #4EC64B;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #4EC64B;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #4EC64B;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>default</td>\n",
       "      <td><div title=\"cac368c7fc33455f97ca806e5c7abf2f\"><a href=\"https://dashboard.default-tenant.app.yh38.iguazio-cd2.com/mlprojects/default/jobs/monitor/cac368c7fc33455f97ca806e5c7abf2f/overview\" target=\"_blank\" >...5c7abf2f</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Oct 23 12:47:45</td>\n",
       "      <td>completed</td>\n",
       "      <td>random-demo</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=admin</div><div class=\"dictlist\">kind=handler</div><div class=\"dictlist\">owner=admin</div></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">best_iteration=4</div><div class=\"dictlist\">multiplier=60</div></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result5d3a14cb\" title=\"files/v3io/projects/default/artifacts/iteration_results.csv\">iteration_results</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"result5d3a14cb-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"result5d3a14cb-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"result5d3a14cb\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"result5d3a14cb-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<b> > to track results use the .show() or .logs() methods  or <a href=\"https://dashboard.default-tenant.app.yh38.iguazio-cd2.com/mlprojects/default/jobs/monitor/cac368c7fc33455f97ca806e5c7abf2f/overview\" target=\"_blank\">click here</a> to open in UI</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2021-10-23 12:47:46,177 [info] run executed, status=completed\n"
     ]
    }
   ],
   "source": [
    "grid_params = {\"p1\": [2, 4, 1, 3], \"p2\": [10, 20, 30]}\n",
    "task = mlrun.new_task(\"random-demo\")\n",
    "task.with_hyper_params(\n",
    "    grid_params, selector=\"max.multiplier\", strategy=\"random\", max_iterations=4\n",
    ")\n",
    "run = mlrun.new_function().run(task, handler=hyper_func)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List search\n",
    "\n",
    "This example also shows how to use the `stop_condition` option."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2021-10-23 12:47:46,184 [info] starting run list-demo uid=136edfb9c9404a61933c73bbbd35b18b DB=http://mlrun-api:8080\n",
      "p1=2, p2=15, result=30\n",
      "p1=3, p2=10, result=30\n",
      "p1=7, p2=10, result=70\n",
      "> 2021-10-23 12:47:47,193 [info] reached early stop condition (multiplier>=70), stopping iterations!\n",
      "> 2021-10-23 12:47:47,195 [info] best iteration=3, used criteria max.multiplier\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #4EC64B;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #4EC64B;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #4EC64B;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>default</td>\n",
       "      <td><div title=\"136edfb9c9404a61933c73bbbd35b18b\"><a href=\"https://dashboard.default-tenant.app.yh38.iguazio-cd2.com/mlprojects/default/jobs/monitor/136edfb9c9404a61933c73bbbd35b18b/overview\" target=\"_blank\" >...bd35b18b</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Oct 23 12:47:46</td>\n",
       "      <td>completed</td>\n",
       "      <td>list-demo</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=admin</div><div class=\"dictlist\">kind=handler</div><div class=\"dictlist\">owner=admin</div></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">best_iteration=3</div><div class=\"dictlist\">multiplier=70</div></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result76e1b9ba\" title=\"files/v3io/projects/default/artifacts/iteration_results.csv\">iteration_results</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"result76e1b9ba-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"result76e1b9ba-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"result76e1b9ba\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"result76e1b9ba-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<b> > to track results use the .show() or .logs() methods  or <a href=\"https://dashboard.default-tenant.app.yh38.iguazio-cd2.com/mlprojects/default/jobs/monitor/136edfb9c9404a61933c73bbbd35b18b/overview\" target=\"_blank\">click here</a> to open in UI</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2021-10-23 12:47:47,385 [info] run executed, status=completed\n"
     ]
    }
   ],
   "source": [
    "list_params = {\"p1\": [2, 3, 7, 4, 5], \"p2\": [15, 10, 10, 20, 30]}\n",
    "task = mlrun.new_task(\"list-demo\").with_hyper_params(\n",
    "    list_params,\n",
    "    selector=\"max.multiplier\",\n",
    "    strategy=\"list\",\n",
    "    stop_condition=\"multiplier>=70\",\n",
    ")\n",
    "run = mlrun.new_function().run(task, handler=hyper_func)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Custom iterator\n",
    "\n",
    "You can define a child iteration context under the parent/main run. The child run is logged independently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def handler(context: mlrun.MLClientCtx, param_list):\n",
    "    best_multiplier = total = 0\n",
    "    for param in param_list:\n",
    "        with context.get_child_context(**param) as child:\n",
    "            hyper_func(child, **child.parameters)\n",
    "            multiplier = child.results[\"multiplier\"]\n",
    "            total += multiplier\n",
    "            if multiplier > best_multiplier:\n",
    "                child.mark_as_best()\n",
    "                best_multiplier = multiplier\n",
    "\n",
    "    # log result at the parent\n",
    "    context.log_result(\"avg_multiplier\", total / len(param_list))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2021-10-23 12:47:47,403 [info] starting run mlrun-a79c5c-handler uid=c3eb08ebae02464ca4025c77b12e3c39 DB=http://mlrun-api:8080\n",
      "p1=2, p2=10, result=20\n",
      "p1=3, p2=30, result=90\n",
      "p1=4, p2=7, result=28\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #4EC64B;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #4EC64B;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #4EC64B;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>default</td>\n",
       "      <td><div title=\"c3eb08ebae02464ca4025c77b12e3c39\"><a href=\"https://dashboard.default-tenant.app.yh38.iguazio-cd2.com/mlprojects/default/jobs/monitor/c3eb08ebae02464ca4025c77b12e3c39/overview\" target=\"_blank\" >...b12e3c39</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Oct 23 12:47:47</td>\n",
       "      <td>completed</td>\n",
       "      <td>mlrun-a79c5c-handler</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=admin</div><div class=\"dictlist\">kind=handler</div><div class=\"dictlist\">owner=admin</div><div class=\"dictlist\">host=jupyter-6476bb5f85-bjc4m</div></td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">param_list=[{'p1': 2, 'p2': 10}, {'p1': 3, 'p2': 30}, {'p1': 4, 'p2': 7}]</div></td>\n",
       "      <td><div class=\"dictlist\">best_iteration=2</div><div class=\"dictlist\">multiplier=90</div><div class=\"dictlist\">avg_multiplier=46.0</div></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"result50355e5c-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"result50355e5c-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"result50355e5c\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"result50355e5c-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<b> > to track results use the .show() or .logs() methods  or <a href=\"https://dashboard.default-tenant.app.yh38.iguazio-cd2.com/mlprojects/default/jobs/monitor/c3eb08ebae02464ca4025c77b12e3c39/overview\" target=\"_blank\">click here</a> to open in UI</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2021-10-23 12:47:48,734 [info] run executed, status=completed\n"
     ]
    }
   ],
   "source": [
    "param_list = [{\"p1\": 2, \"p2\": 10}, {\"p1\": 3, \"p2\": 30}, {\"p1\": 4, \"p2\": 7}]\n",
    "run = mlrun.new_function().run(handler=handler, params={\"param_list\": param_list})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parallel execution over containers\n",
    "\n",
    "When working with compute intensive or long running tasks you'll want to run your iterations over a cluster of containers. At the same time, you don't want to bring up too many containers, and you want to limit the number of parallel tasks.\n",
    "\n",
    "MLRun supports distribution of the child runs over Dask or Nuclio clusters. This is handled automatically by MLRun. You only need to deploy the Dask or Nuclio function used by the workers, and set the level of parallelism in the task. The execution can be controlled from the client/notebook, or can have a job (immediate or scheduled) that controls the execution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Code example (single task)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# mark the start of a code section that will be sent to the job\n",
    "# mlrun: start-code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import socket\n",
    "import pandas as pd\n",
    "\n",
    "\n",
    "def hyper_func2(context, data, p1, p2, p3):\n",
    "    print(data.as_df().head())\n",
    "    context.logger.info(f\"p2={p2}, p3={p3}, r1={p2 * p3} at {socket.gethostname()}\")\n",
    "    context.log_result(\"r1\", p2 * p3)\n",
    "    raw_data = {\n",
    "        \"first_name\": [\"Jason\", \"Molly\", \"Tina\", \"Jake\", \"Amy\"],\n",
    "        \"age\": [42, 52, 36, 24, 73],\n",
    "        \"testScore\": [25, 94, 57, 62, 70],\n",
    "    }\n",
    "    df = pd.DataFrame(raw_data, columns=[\"first_name\", \"age\", \"testScore\"])\n",
    "    context.log_dataset(\"mydf\", df=df, stats=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# mlrun: end-code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running the workers using Dask\n",
    "\n",
    "This example creates a new function and executes the parent/controller as an MLRun `job` and the different child runs over a Dask cluster (MLRun Dask function)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define a Dask cluster (using MLRun serverless Dask)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'db://default/dask-cluster'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dask_cluster = mlrun.new_function(\"dask-cluster\", kind=\"dask\", image=\"mlrun/mlrun\")\n",
    "dask_cluster.apply(mlrun.mount_v3io())  # add volume mounts\n",
    "dask_cluster.spec.service_type = \"NodePort\"  # open interface to the dask UI dashboard\n",
    "dask_cluster.spec.replicas = 2  # define two containers\n",
    "uri = dask_cluster.save()\n",
    "uri"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2021-10-23 12:48:49,020 [info] trying dask client at: tcp://mlrun-dask-cluster-eea516ff-5.default-tenant:8786\n",
      "> 2021-10-23 12:48:49,049 [info] using remote dask scheduler (mlrun-dask-cluster-eea516ff-5) at: tcp://mlrun-dask-cluster-eea516ff-5.default-tenant:8786\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Mismatched versions found\n",
      "\n",
      "+-------------+--------+-----------+---------+\n",
      "| Package     | client | scheduler | workers |\n",
      "+-------------+--------+-----------+---------+\n",
      "| blosc       | 1.7.0  | 1.10.6    | None    |\n",
      "| cloudpickle | 1.6.0  | 2.0.0     | None    |\n",
      "| distributed | 2.30.0 | 2.30.1    | None    |\n",
      "| lz4         | 3.1.0  | 3.1.3     | None    |\n",
      "| msgpack     | 1.0.0  | 1.0.2     | None    |\n",
      "| tornado     | 6.0.4  | 6.1       | None    |\n",
      "+-------------+--------+-----------+---------+\n",
      "Notes: \n",
      "-  msgpack: Variation is ok, as long as everything is above 0.6\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<a href=\"http://default-tenant.app.yh38.iguazio-cd2.com:31350/status\" target=\"_blank\" >dashboard link: default-tenant.app.yh38.iguazio-cd2.com:31350</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table style=\"border: 2px solid white;\">\n",
       "<tr>\n",
       "<td style=\"vertical-align: top; border: 0px solid white\">\n",
       "<h3 style=\"text-align: left;\">Client</h3>\n",
       "<ul style=\"text-align: left; list-style: none; margin: 0; padding: 0;\">\n",
       "  <li><b>Scheduler: </b>tcp://mlrun-dask-cluster-eea516ff-5.default-tenant:8786</li>\n",
       "  <li><b>Dashboard: </b><a href='http://mlrun-dask-cluster-eea516ff-5.default-tenant:8787/status' target='_blank'>http://mlrun-dask-cluster-eea516ff-5.default-tenant:8787/status</a></li>\n",
       "</ul>\n",
       "</td>\n",
       "<td style=\"vertical-align: top; border: 0px solid white\">\n",
       "<h3 style=\"text-align: left;\">Cluster</h3>\n",
       "<ul style=\"text-align: left; list-style:none; margin: 0; padding: 0;\">\n",
       "  <li><b>Workers: </b>0</li>\n",
       "  <li><b>Cores: </b>0</li>\n",
       "  <li><b>Memory: </b>0 B</li>\n",
       "</ul>\n",
       "</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<Client: 'tcp://10.200.0.72:8786' processes=0 threads=0, memory=0 B>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# initialize the dask cluster and get its dashboard url\n",
    "dask_cluster.client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define the parallel work\n",
    "\n",
    "Set the `parallel_runs` attribute to indicate how many child tasks to run in parallel. Set the `dask_cluster_uri` to point \n",
    "to the dask cluster (if it's not set the cluster uri uses dask local). You can also set the `teardown_dask` flag to free up \n",
    "all the dask resources after completion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<mlrun.model.RunTemplate at 0x7f673d7b1910>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_params = {\"p2\": [2, 1, 4, 1], \"p3\": [10, 20]}\n",
    "task = mlrun.new_task(\n",
    "    params={\"p1\": 8},\n",
    "    inputs={\"data\": \"https://s3.wasabisys.com/iguazio/data/iris/iris_dataset.csv\"},\n",
    ")\n",
    "task.with_hyper_params(\n",
    "    grid_params,\n",
    "    selector=\"r1\",\n",
    "    strategy=\"grid\",\n",
    "    parallel_runs=4,\n",
    "    dask_cluster_uri=uri,\n",
    "    teardown_dask=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Define a job that will take the code (using `code_to_function`) and run it over the cluster**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "fn = mlrun.code_to_function(name=\"hyper-tst\", kind=\"job\", image=\"mlrun/mlrun\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2021-10-23 12:49:56,388 [info] starting run hyper-tst-hyper_func2 uid=50eb72f5b0734954b8b1c57494f325bc DB=http://mlrun-api:8080\n",
      "> 2021-10-23 12:49:56,565 [info] Job is running in the background, pod: hyper-tst-hyper-func2-9g6z8\n",
      "> 2021-10-23 12:49:59,813 [info] trying dask client at: tcp://mlrun-dask-cluster-eea516ff-5.default-tenant:8786\n",
      "> 2021-10-23 12:50:09,828 [warning] remote scheduler at tcp://mlrun-dask-cluster-eea516ff-5.default-tenant:8786 not ready, will try to restart Timed out trying to connect to tcp://mlrun-dask-cluster-eea516ff-5.default-tenant:8786 after 10 s\n",
      "> 2021-10-23 12:50:15,733 [info] using remote dask scheduler (mlrun-dask-cluster-04574796-5) at: tcp://mlrun-dask-cluster-04574796-5.default-tenant:8786\n",
      "remote dashboard: default-tenant.app.yh38.iguazio-cd2.com:32577\n",
      "> --------------- Iteration: (1) ---------------\n",
      "   sepal length (cm)  sepal width (cm)  ...  petal width (cm)  label\n",
      "0                5.1               3.5  ...               0.2      0\n",
      "1                4.9               3.0  ...               0.2      0\n",
      "2                4.7               3.2  ...               0.2      0\n",
      "3                4.6               3.1  ...               0.2      0\n",
      "4                5.0               3.6  ...               0.2      0\n",
      "\n",
      "[5 rows x 5 columns]\n",
      "> 2021-10-23 12:50:21,353 [info] p2=2, p3=10, r1=20 at mlrun-dask-cluster-04574796-5k5lhq\n",
      "\n",
      "> --------------- Iteration: (3) ---------------\n",
      "   sepal length (cm)  sepal width (cm)  ...  petal width (cm)  label\n",
      "0                5.1               3.5  ...               0.2      0\n",
      "1                4.9               3.0  ...               0.2      0\n",
      "2                4.7               3.2  ...               0.2      0\n",
      "3                4.6               3.1  ...               0.2      0\n",
      "4                5.0               3.6  ...               0.2      0\n",
      "\n",
      "[5 rows x 5 columns]\n",
      "> 2021-10-23 12:50:21,459 [info] p2=4, p3=10, r1=40 at mlrun-dask-cluster-04574796-5k5lhq\n",
      "\n",
      "> --------------- Iteration: (4) ---------------\n",
      "   sepal length (cm)  sepal width (cm)  ...  petal width (cm)  label\n",
      "0                5.1               3.5  ...               0.2      0\n",
      "1                4.9               3.0  ...               0.2      0\n",
      "2                4.7               3.2  ...               0.2      0\n",
      "3                4.6               3.1  ...               0.2      0\n",
      "4                5.0               3.6  ...               0.2      0\n",
      "\n",
      "[5 rows x 5 columns]\n",
      "> 2021-10-23 12:50:21,542 [info] p2=1, p3=10, r1=10 at mlrun-dask-cluster-04574796-5k5lhq\n",
      "\n",
      "> --------------- Iteration: (6) ---------------\n",
      "   sepal length (cm)  sepal width (cm)  ...  petal width (cm)  label\n",
      "0                5.1               3.5  ...               0.2      0\n",
      "1                4.9               3.0  ...               0.2      0\n",
      "2                4.7               3.2  ...               0.2      0\n",
      "3                4.6               3.1  ...               0.2      0\n",
      "4                5.0               3.6  ...               0.2      0\n",
      "\n",
      "[5 rows x 5 columns]\n",
      "> 2021-10-23 12:50:21,629 [info] p2=1, p3=20, r1=20 at mlrun-dask-cluster-04574796-5k5lhq\n",
      "\n",
      "> --------------- Iteration: (7) ---------------\n",
      "   sepal length (cm)  sepal width (cm)  ...  petal width (cm)  label\n",
      "0                5.1               3.5  ...               0.2      0\n",
      "1                4.9               3.0  ...               0.2      0\n",
      "2                4.7               3.2  ...               0.2      0\n",
      "3                4.6               3.1  ...               0.2      0\n",
      "4                5.0               3.6  ...               0.2      0\n",
      "\n",
      "[5 rows x 5 columns]\n",
      "> 2021-10-23 12:50:21,792 [info] p2=4, p3=20, r1=80 at mlrun-dask-cluster-04574796-5k5lhq\n",
      "\n",
      "> --------------- Iteration: (8) ---------------\n",
      "   sepal length (cm)  sepal width (cm)  ...  petal width (cm)  label\n",
      "0                5.1               3.5  ...               0.2      0\n",
      "1                4.9               3.0  ...               0.2      0\n",
      "2                4.7               3.2  ...               0.2      0\n",
      "3                4.6               3.1  ...               0.2      0\n",
      "4                5.0               3.6  ...               0.2      0\n",
      "\n",
      "[5 rows x 5 columns]\n",
      "> 2021-10-23 12:50:22,052 [info] p2=1, p3=20, r1=20 at mlrun-dask-cluster-04574796-5k5lhq\n",
      "\n",
      "> --------------- Iteration: (2) ---------------\n",
      "   sepal length (cm)  sepal width (cm)  ...  petal width (cm)  label\n",
      "0                5.1               3.5  ...               0.2      0\n",
      "1                4.9               3.0  ...               0.2      0\n",
      "2                4.7               3.2  ...               0.2      0\n",
      "3                4.6               3.1  ...               0.2      0\n",
      "4                5.0               3.6  ...               0.2      0\n",
      "\n",
      "[5 rows x 5 columns]\n",
      "> 2021-10-23 12:50:23,134 [info] p2=1, p3=10, r1=10 at mlrun-dask-cluster-04574796-5j6v59\n",
      "\n",
      "> --------------- Iteration: (5) ---------------\n",
      "   sepal length (cm)  sepal width (cm)  ...  petal width (cm)  label\n",
      "0                5.1               3.5  ...               0.2      0\n",
      "1                4.9               3.0  ...               0.2      0\n",
      "2                4.7               3.2  ...               0.2      0\n",
      "3                4.6               3.1  ...               0.2      0\n",
      "4                5.0               3.6  ...               0.2      0\n",
      "\n",
      "[5 rows x 5 columns]\n",
      "> 2021-10-23 12:50:23,219 [info] p2=2, p3=20, r1=40 at mlrun-dask-cluster-04574796-5k5lhq\n",
      "\n",
      "> 2021-10-23 12:50:23,261 [info] tearing down the dask cluster..\n",
      "> 2021-10-23 12:50:43,363 [info] best iteration=7, used criteria r1\n",
      "> 2021-10-23 12:50:43,626 [info] run executed, status=completed\n",
      "final state: completed\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #4EC64B;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #4EC64B;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #4EC64B;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>default</td>\n",
       "      <td><div title=\"50eb72f5b0734954b8b1c57494f325bc\"><a href=\"https://dashboard.default-tenant.app.yh38.iguazio-cd2.com/mlprojects/default/jobs/monitor/50eb72f5b0734954b8b1c57494f325bc/overview\" target=\"_blank\" >...94f325bc</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Oct 23 12:49:59</td>\n",
       "      <td>completed</td>\n",
       "      <td>hyper-tst-hyper_func2</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=admin</div><div class=\"dictlist\">kind=job</div><div class=\"dictlist\">owner=admin</div></td>\n",
       "      <td><div title=\"https://s3.wasabisys.com/iguazio/data/iris/iris_dataset.csv\">data</div></td>\n",
       "      <td><div class=\"dictlist\">p1=8</div></td>\n",
       "      <td><div class=\"dictlist\">best_iteration=7</div><div class=\"dictlist\">r1=80</div></td>\n",
       "      <td><div title=\"v3io:///projects/default/artifacts/7/mydf\">mydf</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultcdedeefc\" title=\"files/v3io/projects/default/artifacts/iteration_results.csv\">iteration_results</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"resultcdedeefc-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"resultcdedeefc-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"resultcdedeefc\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"resultcdedeefc-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<b> > to track results use the .show() or .logs() methods  or <a href=\"https://dashboard.default-tenant.app.yh38.iguazio-cd2.com/mlprojects/default/jobs/monitor/50eb72f5b0734954b8b1c57494f325bc/overview\" target=\"_blank\">click here</a> to open in UI</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2021-10-23 12:50:53,303 [info] run executed, status=completed\n"
     ]
    }
   ],
   "source": [
    "run = fn.run(task, handler=hyper_func2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running the workers using Nuclio\n",
    "\n",
    "Nuclio is a high-performance serverless engine that can process many events in parallel. It can also separate initialization from execution. Certain parts of the code (imports, loading data, etc.) can be done once per worker vs. in any run.\n",
    "\n",
    "Nuclio, by default, process events (http, stream, ..). There is a special Nuclio kind that runs MLRun jobs (nuclio:mlrun).\n",
    "\n",
    "```{admonition} Notes\n",
    "* Nuclio tasks are relatively short (preferably under 5 minutes), use it for running many iterations where each individual run is less than 5 min.\n",
    "* Use `context.logger` to drive text outputs (vs `print()`).\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create a nuclio:mlrun function\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2021-10-23 12:51:10,152 [info] Starting remote function deploy\n",
      "2021-10-23 12:51:10  (info) Deploying function\n",
      "2021-10-23 12:51:10  (info) Building\n",
      "2021-10-23 12:51:10  (info) Staging files and preparing base images\n",
      "2021-10-23 12:51:10  (info) Building processor image\n",
      "2021-10-23 12:51:11  (info) Build complete\n",
      "2021-10-23 12:51:19  (info) Function deploy complete\n",
      "> 2021-10-23 12:51:22,296 [info] successfully deployed function: {'internal_invocation_urls': ['nuclio-default-hyper-tst2.default-tenant.svc.cluster.local:8080'], 'external_invocation_urls': ['default-tenant.app.yh38.iguazio-cd2.com:32760']}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'http://default-tenant.app.yh38.iguazio-cd2.com:32760'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fn = mlrun.code_to_function(name=\"hyper-tst2\", kind=\"nuclio:mlrun\", image=\"mlrun/mlrun\")\n",
    "# replicas * workers need to match or exceed parallel_runs\n",
    "fn.spec.replicas = 2\n",
    "fn.with_http(workers=2)\n",
    "fn.deploy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run the parallel task over the function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this is required to fix Jupyter issue with asyncio (not required outside of Jupyter)\n",
    "# run it only once\n",
    "import nest_asyncio\n",
    "\n",
    "nest_asyncio.apply()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2021-10-23 12:51:31,618 [info] starting run hyper-tst2-hyper_func2 uid=97cc3e255f3c4c93822b0154d63f47f5 DB=http://mlrun-api:8080\n",
      "> --------------- Iteration: (4) ---------------\n",
      "2021-10-23 12:51:32.130812  info   logging run results to: http://mlrun-api:8080  worker_id=1\n",
      "2021-10-23 12:51:32.401258  info   p2=1, p3=10, r1=10 at nuclio-default-hyper-tst2-5d4976b685-47dh6  worker_id=1\n",
      "\n",
      "> --------------- Iteration: (2) ---------------\n",
      "2021-10-23 12:51:32.130713  info   logging run results to: http://mlrun-api:8080  worker_id=0\n",
      "2021-10-23 12:51:32.409468  info   p2=1, p3=10, r1=10 at nuclio-default-hyper-tst2-5d4976b685-47dh6  worker_id=0\n",
      "\n",
      "> --------------- Iteration: (1) ---------------\n",
      "2021-10-23 12:51:32.130765  info   logging run results to: http://mlrun-api:8080  worker_id=0\n",
      "2021-10-23 12:51:32.432121  info   p2=2, p3=10, r1=20 at nuclio-default-hyper-tst2-5d4976b685-2gdtc  worker_id=0\n",
      "\n",
      "> --------------- Iteration: (5) ---------------\n",
      "2021-10-23 12:51:32.568848  info   logging run results to: http://mlrun-api:8080  worker_id=0\n",
      "2021-10-23 12:51:32.716415  info   p2=2, p3=20, r1=40 at nuclio-default-hyper-tst2-5d4976b685-47dh6  worker_id=0\n",
      "\n",
      "> --------------- Iteration: (7) ---------------\n",
      "2021-10-23 12:51:32.855399  info   logging run results to: http://mlrun-api:8080  worker_id=1\n",
      "2021-10-23 12:51:33.054417  info   p2=4, p3=20, r1=80 at nuclio-default-hyper-tst2-5d4976b685-2gdtc  worker_id=1\n",
      "\n",
      "> --------------- Iteration: (6) ---------------\n",
      "2021-10-23 12:51:32.970002  info   logging run results to: http://mlrun-api:8080  worker_id=0\n",
      "2021-10-23 12:51:33.136621  info   p2=1, p3=20, r1=20 at nuclio-default-hyper-tst2-5d4976b685-47dh6  worker_id=0\n",
      "\n",
      "> --------------- Iteration: (3) ---------------\n",
      "2021-10-23 12:51:32.541187  info   logging run results to: http://mlrun-api:8080  worker_id=1\n",
      "2021-10-23 12:51:33.301200  info   p2=4, p3=10, r1=40 at nuclio-default-hyper-tst2-5d4976b685-47dh6  worker_id=1\n",
      "\n",
      "> --------------- Iteration: (8) ---------------\n",
      "2021-10-23 12:51:33.419442  info   logging run results to: http://mlrun-api:8080  worker_id=0\n",
      "2021-10-23 12:51:33.672165  info   p2=1, p3=20, r1=20 at nuclio-default-hyper-tst2-5d4976b685-47dh6  worker_id=0\n",
      "\n",
      "> 2021-10-23 12:51:34,153 [info] best iteration=7, used criteria r1\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #4EC64B;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #4EC64B;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #4EC64B;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>default</td>\n",
       "      <td><div title=\"97cc3e255f3c4c93822b0154d63f47f5\"><a href=\"https://dashboard.default-tenant.app.yh38.iguazio-cd2.com/mlprojects/default/jobs/monitor/97cc3e255f3c4c93822b0154d63f47f5/overview\" target=\"_blank\" >...d63f47f5</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Oct 23 12:51:31</td>\n",
       "      <td>completed</td>\n",
       "      <td>hyper-tst2-hyper_func2</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=admin</div><div class=\"dictlist\">kind=remote</div><div class=\"dictlist\">owner=admin</div></td>\n",
       "      <td><div title=\"https://s3.wasabisys.com/iguazio/data/iris/iris_dataset.csv\">data</div></td>\n",
       "      <td><div class=\"dictlist\">p1=8</div></td>\n",
       "      <td><div class=\"dictlist\">best_iteration=7</div><div class=\"dictlist\">r1=80</div></td>\n",
       "      <td><div title=\"v3io:///projects/default/artifacts/7/mydf\">mydf</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result7f47fd92\" title=\"files/v3io/projects/default/artifacts/iteration_results.csv\">iteration_results</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"result7f47fd92-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"result7f47fd92-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"result7f47fd92\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"result7f47fd92-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<b> > to track results use the .show() or .logs() methods  or <a href=\"https://dashboard.default-tenant.app.yh38.iguazio-cd2.com/mlprojects/default/jobs/monitor/97cc3e255f3c4c93822b0154d63f47f5/overview\" target=\"_blank\">click here</a> to open in UI</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2021-10-23 12:51:34,420 [info] run executed, status=completed\n"
     ]
    }
   ],
   "source": [
    "grid_params = {\"p2\": [2, 1, 4, 1], \"p3\": [10, 20]}\n",
    "task = mlrun.new_task(\n",
    "    params={\"p1\": 8},\n",
    "    inputs={\"data\": \"https://s3.wasabisys.com/iguazio/data/iris/iris_dataset.csv\"},\n",
    ")\n",
    "task.with_hyper_params(\n",
    "    grid_params, selector=\"r1\", strategy=\"grid\", parallel_runs=4, max_errors=3\n",
    ")\n",
    "run = fn.run(task, handler=hyper_func2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
